recorder: inline mode, so a host can have the rows without keeping the bytes - #95

Inline mode derives rows from datagrams it does not keep. Archive mode derives
them from an object whose sha256 was checked against its manifest first, and
whose bytes are still there to derive again. Both write into the same five
tables, and without this column a query cannot tell one from the other.

A column rather than an inference. An inline row's object_sha256 is empty, so a
reader could in principle test for that — which is the reason not to: an empty
digest field is a trap, a query that reads it as verified is wrong in the
direction that matters, and nothing about the name object_sha256 warns anybody.

DeriveInput carries it and has no Default, so a derivation names its provenance
or does not compile, and every row in a batch is stamped from that one value
rather than filled in per grain. derive_object states `archive` because it
verified the digest itself three lines earlier.

In no ORDER BY, on any table, and there is now a test that says so: provenance
in a sort key stops the two modes' views of one datagram collapsing under
ReplacingMergeTree, so a window loaded both ways doubles and every count over
it is wrong.

001 declares the column and 005 adds it by ALTER. 001 is the authoritative
table definition — tests/ddl.rs holds the row types against its CREATE TABLE
blocks column for column — so a fresh deployment gets it from there, and 005 is
for the deployments whose tables already exist, where CREATE TABLE IF NOT
EXISTS reaches nothing. Adding a column with a DEFAULT is a metadata change, so
this is not a mutation of the hundred-million-rows-a-day table.

004 grants INSERT at table level, not per column, so the loader account needs no
further grant.

Mechanical, and nothing changed behaviour. `cargo test -p dz-recorder-load
-- --list` names the same 56 tests before and after, which is the check this
kind of move deserves: passing tests only say the code still works, and an
identical test set says nothing moved out of reach of one.

ledger, loader and metrics go into src/lib.rs. Inline mode captures and derives
in one process, so it has no objects directory to walk and does not want
Loader — but everything downstream of the derivation is the same problem, and a
second ledger would be a second answer to *is this loaded* that disagrees with
the first in exactly the case that matters: after a crash. record_landed
carries the rule that an entry is written when an insert is acknowledged and
never when the sink accepted the rows, and that rule is worth having once.

The command line, the configuration, the endpoint and the build identity stay
in the binary. Each is a statement about this binary — inline mode reads its own
configuration, serves its own endpoint on the recorder's port and reports the
recorder's build — so exporting them would hand a caller the wrong one.

Recorded, record_landed and now_unix_nanos were pub(crate), which reaches
neither the binary nor inline mode once the modules live in a library: two
targets are two crates.

Ten pull requests landed while this branch was in progress, and one of them —
the market data rows work — added three grains, three tables and three
migrations. So this is not a mechanical merge, and the parts that were not
mechanical are these.

`derivation` now covers eight grains, not five. `Event`, `Instrument` and
`BookTop` are derived from the same datagrams by the same pass, so a reader
joining an event row back to the datagram row it came from has to find the same
answer on both; a provenance that stopped at the five original tables would be a
column that stops being true exactly where the join gets interesting.
`EventInput` carries the field for the same reason `DeriveInput` does — the fold
reads a `Source` and cannot see whether the bytes behind it were verified.

The migration is `008`. `005` through `007` were taken by market data while this
branch held `005`, and the five cross-references in `001` were renumbered with
it.

`the_provenance_column_is_in_no_sort_key` failed on the merged tree, naming the
three grains it could not find, because it read only `001` and those tables are
declared in `005`. That is the test doing its job rather than a test to relax:
it now maps each grain to the migration that declares it, and scans all four
schema files for a sort key that mentions the column.

The two golden provenance tests enumerated five vectors by hand and would have
passed vacuously over the new three. They now go through an exhaustive `match`
on `Grain`, so the next grain fails to compile here instead of shipping rows
nobody can attribute — the same discipline the column-store sink already applies
when it destructures a batch.

The library extraction had to take `config` and `market_data` with it. The pass
now reaches into both, and a pass cannot be a library while half of what it
calls is not. `lib.rs` says so rather than leaving the boundary looking arbitrary.

`rust/Cargo.lock` and four import lists conflicted textually and were resolved
by keeping both sides. The whole workspace builds, formats, lints and passes 170
test suites on the merged tree.

It lived in the replay crate because the two callers that needed it were an
archive iterator and a round-trip test. Inline mode is a third: a live capture
handing datagrams across a thread boundary owns what it carries, and reaching
this type through the archive reader would pull a pcapng parser and a
decompressor into a path that reads no archive at all.

`dz-recorder-replay` re-exports it, so every existing caller reaches it where it
always has. 174 test suites unchanged.

…ntly

Cargo.toml, lib.rs and four empty modules. dz-recorder-replay is deliberately
not a dependency: inline mode reads no archive, and OwnedDatagram now lives in
core precisely so this build carries no pcapng parser and no decompressor.

Inline mode puts a derivation behind a live capture, so there is now a place a
datagram can be lost that did not exist before. A datagram the deriver never
sees is a sequence value nobody delivered, and a sequence value nobody delivered
with nothing admitted behind it gets a `publisher` verdict — the recorder's own
drop reported as somebody else's fault. That is the failure this module exists
to prevent, and it is worse than not measuring at all.

So a drop is charged through PendingLoss, the accumulator the capture already
uses one layer down. The debt rides on the drop_delta of the next datagram that
gets through, which is what that field is defined as. `offer` owes the incoming
delta to the accumulator BEFORE it tries rather than after it succeeds: the
offer may fail, and a delta that left on a datagram which did not get through is
loss the rows never hear about.

It never blocks. A capture thread waiting for a slot stops draining its receive
queue, and a slow derivation becomes feed loss plus a false publisher finding in
every window derived during it. A full ring is a drop and a counter — the
staging watermark's rule, applied to datagrams.

The slots are pooled. OwnedDatagram::from_recorded allocates, and an allocation
per datagram on the capture thread is a regression against a path that costs a
copy and a buffered write today. The test asserts it on the buffer addresses the
derivation is handed: 200 datagrams through a ring of 4 must come back in at
most 4 distinct buffers.

One slot is always in the deriver's hand, so a producer sees capacity - 1 free
while a datagram is out. That is what makes the borrow safe rather than an
off-by-one, and it is written down where a reader will hit it.

…claim

The window is the object's replacement and only the object's. `derive` reads a
Source to exhaustion and consults one thing outside the window it is given — the
preceding trailer, which decides one bit — so the object was never the unit of
derivation, it was the unit of storage. This makes a live capture look like a
source that ends.

The bound is checked before the receive, never after. A window that took a
datagram and then found it did not fit would have to hold it over, and a
datagram held between two windows is in neither one's tally. Checked first, it
stays in the ring and opens the next window instead — asserted on the sequence
numbers rather than on a count, so a datagram quietly dropped at the boundary
fails the test.

The wait is sliced, because the age bound exists for a feed that has gone quiet
and a receive that blocked until traffic arrived would never let it fire.

CoverageTracker is the archive writer's own, not a second implementation. Two
accumulators would be two answers to what a window covered, and they would
differ in exactly the cases the coverage row exists to describe: a reset inside
the window, a datagram too short to attribute, an instance past the cap.

The manifest states what was observed and refuses the rest. `sha256` is empty
and `byte_count` is zero because nothing was written and nothing was hashed — a
digest over anything else would be a different claim wearing the field name of a
claim about an archived object, and a reader checking it would be checking
nothing. The window key carries the wall-clock start rather than the sequence
number, which restarts at zero every run, and says `live/` so nobody goes
looking for an object to fetch.

The ring gained a wait/in_hand split: a caller that waits in a loop and then
returns the datagram cannot hold a borrow across the loop, because the borrow
checker cannot see the previous iteration's is dead.

The fixtures are the synthetic publisher's. Hand-built datagrams were tried
first and were the wrong fixture: the coverage tracker reads the channel, the
sequence number and the reset count at fixed offsets, so a test that writes
those offsets itself only tests its own idea of the layout.

One synthetic feed, two paths, the same rows. The argument for deriving in
flight is not that it is cheaper — it is that it is the same function as
deriving from an object. If it is, a row's provenance is the only thing that
changed and every query written against archive-mode rows still means what it
meant. If it is not, inline mode is a second analysis wearing the first one's
column names.

A clean feed and all nine injected faults — a gap, backward motion, a reset, a
second publisher, one disappearing, a duplicate, a reordered pair, an over-cap
declared length, an unknown schema version — now derive identically through both
paths, field for field, on every grain. The comparison erases only `derivation`,
`object_key` and `object_sha256`, and it erases rather than skips: a comparison
that walked the fields it cared about would ignore any field added later, which
is most of this test's value.

It compares grain by grain and row by row. Two whole batches printed on failure
are hundreds of rows with the difference somewhere inside them, and a gate whose
failure nobody can read is a gate that gets deleted.

The third test is the one archive mode has no equivalent of: a ring overrun in
the middle of a feed must not produce a `publisher` verdict. The loss has to
fall between two datagrams that arrived, or nothing reveals it — so the capture
runs where it runs in production, on its own thread, bursting far past the ring
and then pausing, and the datagram that reopens the feed is the one that admits
what was lost. Its gaps come back attributed to the recorder with no unexplained
residue.

Also lands the spool: rows on disk between the derivation and the column store,
one window to a directory, a per-grain digest, fsync once at close, oldest
first, a byte budget that evicts rather than blocks, replay of what a crash
left, and a ledger entry written only after an insert is acknowledged. Eight
tests, all about the failures rather than the happy path.

Inline mode is asked for with --inline-config, and RecorderConfig gains no key.
Its config_hash is written into every archived object as provenance, so a
destination in that file would make a password rotation change what an archive
says produced it — and adding any field changes the hash of every configuration
in the fleet. site, recorder and env are not in the second file either: two
files can name the same host differently, and then one dashboard's live panel
and historical panel describe two recorders that do not exist.

Everything is behind a build feature that is off by default, so the recorder
still builds with no column-store crate and no HTTP client.

The four refusals each name their key. An archive directory configured in inline
mode is refused rather than ignored, because a host believed to be keeping bytes
it never kept for a second is the worst outcome available here. A spool
directory that cannot be opened is refused because the spool is this mode's
whole durability. A ledger inside the spool is refused for the reason the
loader's may not live inside its objects directory. And --inline-config in a
build without the feature fails at startup rather than falling back, because
falling back leaves a host in the arrangement nobody chose.

Two things this leaves open, both stated rather than papered over.

Plan::from_config requires staging_dir and completed_dir to be non-empty, and
inline mode refuses them with a value, so the two are mutually exclusive as
written: --check in inline mode validates the identity, both files and the
destination, but not yet the feeds. Making Plan mode-aware is a refactor of
startup.rs and belongs with the runner.

--inline-config without --check therefore errors saying recording is not wired
yet. Falling back to archive mode was the one thing forbidden.

…posts without a lock

Two structural gaps found by building on top of the pieces, not by reading them.

Plan::from_config requires the archive directories and inline mode refuses them
with a value, so the two were mutually exclusive: --check in inline mode
validated the identity, both files and the destination, but not a single feed.
A group that was not multicast, a port role claimed twice, an interface that did
not resolve — all would have started. On the arrangement that keeps no bytes,
which is the one that leaves an operator least to diagnose with.

So Plan has two constructors and carries which one made it. for_inline runs
every feed check from_config runs — inline mode joins exactly what archive mode
joins — and drops only the archive's: no directories, no compression, no staging
budget, because those keys are refused outright by the other file. FeedPlan's
writer configuration becomes Option, absent in inline mode: a set of directories
built for a mode that never opens a writer is a set of directories an operator
would reasonably expect to find objects in. The archive runner asserts the
arrangement it was handed, so a wiring mistake names itself rather than
surfacing as a panic in a feed thread.

The identity refusal moved to the plan and is now made once. Inline mode takes
site, recorder and env from the recorder's own file, so the check belongs where
that file is validated, and an operator reads the same sentence whichever
arrangement they run.

Arrangement::Inline and for_inline are behind the feature. A default build has
no inline mode to be in, and a variant it can never construct is one its
dead-code analysis is right to flag.

The spool gained a two-phase post. Its `post` held `&mut self` across
sink.write_batch, and under the Arc<Mutex<Spool>> the pipeline needs — a
derivation thread storing, a posting thread posting — that put an HTTP request
inside the lock. A slow destination would then block the store, which blocks the
derivation, which fills the ring, which drops datagrams: the exact backpressure
chain the design forbids, arriving by a route nobody would look for. take_oldest,
record_landed and release let the network call happen with no lock held.
record_landed takes a list because a coalescing sink lands earlier windows
together with the current one.

181 suites green, both builds clean.

The pipeline: a derivation stage turning windows into rows, a spool between,
and a posting stage. Three and not two because write_batch posts synchronously
and retries — on the derivation stage a merely slow destination would hold up
the next window, fill the ring behind it, and turn a column-store problem into
feed loss.

The lock is never held across the network. Both stages reach the spool, so it
sits behind a mutex; the posting stage takes a window under the lock, releases
it, posts, and takes it again to record. That is the whole reason the spool has
a two-phase API.

Writing the tests found two real holes, neither visible from reading the code.

**stop() raced the derivation into abandoning a window.** It set a flag and the
derivation checked it at the top of its loop, so a shutdown arriving before the
first window was open discarded everything already in the ring — datagrams the
capture had accepted and the publisher will not send again, leaving a hole in
the rows that nothing in them could explain. The fix is in the signature: stop
takes the capture end. Closing the ring is what ends the open window, and an
ordering the type system enforces is one no caller can get wrong. The derivation
consults no stop flag at all now.

**A panic between taking a window and recording it stranded it in flight.** The
unwind passed both record_landed and release, leaving a window nobody was
holding that the spool would never offer again — it would sit on disk ageing the
lag gauge until the process restarted. A restarted stage now releases first: a
no-op on the first entry and the recovery on every other. And a stage that
panics while stopping gets two more attempts, because the flag otherwise reads
as "do not come back" and the final flush is exactly when a window is most
likely to be in hand.

post_if_due is called once a pass, as the loader calls it: the sink holds rows
across windows deliberately, so a feed that has gone quiet would hold its last
rows until something else arrived — the opposite of what its age bound is for.
So is record_landed, for the spool's own contract: a window whose rows are in
the store but whose ledger entry would not write owes an entry rather than an
insert, and nothing else retries it.

The recorder README gains a "two modes" section beside the "two capture modes"
one it mirrors — capture mode is orthogonal to the arrangement, and that first
line now says so. It states the cost without softening it: a rule written next
month cannot be run against traffic nobody kept, a derivation bug can be stopped
but not corrected, and nothing verified the bytes the rows came from. Then the
three things that bound it. The loader's README places itself as the other half
of archive mode, BRINGING-UP-A-FEED gains the two files and both --check
invocations, and there is a systemd unit following the loader's own layout.

No README for dz-recorder-inline. Of fifteen crates under rust/recorder only
the binary has one, and the module docs already carry the argument; a second
copy is a copy that drifts.

Three fixes from reading rather than from writing:

scripts/check-public-repo-rules.sh was failing on this tree — a spool test used
10.0.0.1, outside the documentation ranges. That is a gate, and it was red.

The unreachable-destination message named DZ_LOADER_CLICKHOUSE_PASSWORD as the
only source of the password, while the example, the usage text and the config
crate all name DZ_LOADER_CLICKHOUSE_PASSWORD_FILE as the preferred one. An
operator following the error would have taken the second-best route.

A duplicated cfg attribute, harmless and gone.

The metrics endpoint now takes a renderer, so one port can serve two registries:
inline mode publishes dz_recorder_* and dz_recorder_inline_* from one process,
and two ports would be a second target to configure and a second thing to notice
missing.

Left for the record, not fixed here: recorder-release.yml builds --features
afpacket only, so the released asset cannot run inline mode. The loader's
release already carries --features tls, so the precedent exists; whether the
recorder's asset should carry the column-store client is a deployment decision
rather than a wiring one.

The record path, and with it the mode works end to end. A capture thread per
feed offering into a ring the pipeline drains, one metrics port serving both
families, and a shutdown that keeps what the capture is holding.

pump and drain_and_stop are the archive path's own, unchanged. The ordering they
encode — take what the capture already has, then stop it — is the part of this
binary least worth rewriting: both live sources report Ended as soon as their
stop flag is set, so stopping first discards every datagram the drain threads
had queued. Then the pipeline takes the sending end, which is what ends the open
window, so those datagrams are derived rather than abandoned.

A spool and a ledger per feed, for archive mode's own reason. Two writers
sharing a staging directory each see the other's open segment as an orphan and
evict it; two pipelines sharing a spool would do the same under a budget neither
could account for, and two appending to one ledger would interleave lines into a
file that parses as neither. The host's spool budget is divided between the
feeds that share the disk, exactly as the staging budget is.

The signal handler is now one function used by both arrangements rather than two
copies that would drift — including the second-signal exit, which matters more
here: the shutdown waits on a destination that can hang for as long as it likes.

dz_recorder_inline_* covers the three places this arrangement can be wrong that
the other cannot: the ring, the spool, and a stage that stopped. Its own family
beside the health tier's, because folding them together would make an
archive-mode dashboard show empty panels for series that cannot exist there. The
help text on the eviction counter says not to alert on it and names the age
gauge instead.

The roles a window declares are built from the bindings, not copied from a
writer configuration an inline plan does not have. The membership address is
written only when the join named one: route discovery means the kernel picked
it, and an unobserved value must not become a written one.

182 suites green, both builds clean, and the public-repo check passes.

Running the binary rather than the tests turned up two things in what a
deployment pipeline reads.

The identity was printed twice — once by the plan and once by the inline
summary. Two `site=` lines have an operator working out whether the two files
disagree, and the answer is that they cannot: inline mode takes the identity
from the recorder's own file and the second file has no key for it. A summary
that repeated it invented a question the configuration makes unaskable.

And the plan printed `archive rotate=...`, `staging budget=0B per feed` and
`staging=- completed=-` in a mode that writes no object. The keys are refused
outright, so those lines had an operator reading `--check` for a problem that is
the absence of a thing this arrangement does not do — and a staging budget of
zero reads like a misconfiguration rather than like nothing being staged.

Both are now tested rather than left to the next person to notice.

…honest

The default decides how a configuration that says nothing is read, and this
inverts it: inline mode is the reading, archive mode is asked for by
`--archive`.

The argument is not the flag. It is that archive mode requires two directories
inline mode refuses a value for, so an archive-mode host whose command line
names no mode is refused at startup by key rather than read as a recorder that
quietly stopped keeping bytes. There is no third configuration shape. A log
line would not have been enough: the restart that changes a host's mode is the
moment nobody is reading its log.

Three new sections. Why the default belongs on the reading that fails loudly
rather than the one that fails as an empty table. What it costs — a fleet-wide
edit to every archive-mode command line, classed the way the feed-routes design
classes its own, and the build feature moving into the default set because a
binary cannot refuse its own default mode. And whether the default should keep
the bytes too: decided inline only, rejected for four reasons, the strongest
being that a both-mode has to accept the very keys whose refusal makes this
default safe.

`--archive` mints no vocabulary: it is the `[archive]` section, the
`derivation` column's value, the archive crate's name and this design's own
word for the arrangement. Inferring the mode from whether those directories
carry a value is rejected as the same trap the `derivation` column exists to
avoid, one layer up.

…nverts the default

Task 11: the ten places the old default was written down, the two refusals that
have to name the flag, and the revert that has to kill each of the four tests.
Its argument is the spec's; this is the list.

The progress marks are verified against the tree rather than against the
branch's own commit subjects, and four boxes stay open because grepping for
what they name found nothing:

- the window manifest's cumulative capture drop totals. `window_manifest`
  takes them and the only caller passes zeros, so every inline coverage row
  reports a capture that dropped nothing. The equivalence gate cannot see it:
  the synthetic feed has no kernel drops, so both paths report zero and agree.
- the trailer is never read back from the ledger. It is written, and a run
  starts at `preceding: None`, so the era anchor the design says stays certain
  across a restart is uncertain on the first window after every one.
- rows derived has no `grain` label, and there is no readable last error where
  archive mode publishes one.
- `Fault::SilentChannel` is the one of nine the gate does not run, and it is
  the fault whose inline behaviour is least like archive mode's: a channel that
  goes quiet is found by a window closing on age, which is inline mode's own
  key.

Three bullets were amended before being ticked, each with the reason: the
refusals live in `inline_config.rs` and not `startup.rs`; `serve_rendering`
was added beside `serve` rather than changing its signature, which is what *no
change to archive mode* asked for; and the gate erases three provenance fields
and not four, because `byte_count` is a manifest field that no grain carries.

Task 9's three binary-altitude tests are not written and are named as the
largest gap on the branch. Acceptance criteria 2 and 4 need a host rather than
a test and are recorded as open.

The default moves onto the reading that fails loudly. `--archive` is how the
arrangement that keeps every datagram is asked for; its absence is inline mode.

What holds it is not the flag. Archive mode requires `archive.staging_dir` and
`completed_dir`, and inline mode refuses either of them carrying a value, so an
archive-mode host whose command line has not changed is refused at startup by
key and told to pass `--archive` — that message is the migration instruction as
much as the refusal, and it is the only place the instruction is certain to be
read. A configuration stating neither shape is refused naming both flags:
`InlineConfigError::NotStated`, because there is no defensible spool directory,
ledger or destination to invent and a recorder that guessed one would load rows
into a database nobody chose. The archive-directory refusal runs first, before
the missing-file one, because the command line most likely to arrive here by
mistake is an archive-mode host's and the useful answer names the key it wrote.

`inline` joins the default feature set. A binary that refused the arrangement
its own command line asks for when told nothing would be a binary whose default
it cannot honour. `--no-default-features` is now the record-only build — no
column-store client, no HTTP client, no row crates — and it can only be in
archive mode, so it refuses a command line naming no mode by the feature's name
and by `--archive`. What kept the destination out of the record path was never
this feature: it is that the capture path never blocks and never parses, and
that derivation and posting are off it entirely.

Ten places said the old default and all ten moved: the CLI usage and its Args,
the dispatch in main, three error messages, `Arrangement`, the manifest, the
recorder and loader READMEs, the bring-up guide and its checklist, the example
configuration, the systemd unit, and the inline crate's own module doc.

Two CI consequences, both wired. The record-only build gets its own clippy and
test step, because the feature moving into the default set makes it the only
build that compiles the `NotCompiledIn` refusal, and a refusal exercised only
where it is never made is no refusal at all. And the released asset now carries
the mode for free, features being additive over `--features afpacket` — which
settles a deployment question this branch had left open rather than leaving a
released binary that cannot run its own default.

Also fixes a banned word in a workflow comment: `arm` is out in every sense,
and the afpacket cfg blocks are branches.

Five reverts run, each watched fail and restored. Two are worth more than a
table row.

The defaulted-`InlineConfig` revert does not make the recorder start — it gets
as far as an empty `inline.spool_dir` and refuses there. So the mutant is
unhelpful rather than silent, and the test asserts the difference: an operator
whose command line forgot `--archive` would otherwise be told about a key they
never wrote, in a file they never made, on a host they believed was keeping
bytes.

The default-feature revert is caught by nothing but the manifest assertion. With
`inline` out of the default set the whole feature-gated suite is compiled out
rather than failed, and the archive-refusal test still passes through its
`cfg!(not(feature))` branch. A `cfg!` test disappears in the build it exists to
catch, and a suite that shrinks reports success.

Also records two things the task needed that its bullets had not predicted: the
archive-directory refusal has to run before the missing-file one, and every
check in `tests/check_mode.rs` needed the flag — which is this change's
fleet-wide cost in miniature, and why the refusal test belongs in that file.

The eleventh place the default was expressed, and the one the inversion's own
list missed. `tests/shutdown.rs` spawns the real binary against an archive-mode
fixture and waits for a segment to appear in `staging`; read as inline mode it
was refused by key on `archive.staging_dir` before binding a socket, so a
0.7-second test became a fifteen-second timeout. It gets `--archive`, because an
archive-mode shutdown test that quietly became an inline-mode one would be
asserting nothing: the object it waits for is the only thing it proves, and
inline mode writes none.

The refusal firing there is the refusal working.

`cargo test --workspace` did not catch it and cannot: the suite is
`#![cfg(feature = "socket-e2e")]`, and a suite behind a feature reports *0
tests* rather than *skipped*, so a green workspace run says nothing about it.
The plan now records that under the gap it already had about task 9's
binary-altitude tests — same class — with the list of feature combinations to
run before pushing.

Swept every other place that spawns the binary or writes a recorder
configuration: `check_mode.rs` and `inline_mode.rs` already name their modes,
the only recorder systemd unit is the inline one and passes
`--inline-config`, `dz-recorder-load`'s binary test and unit have no mode to
name, and the demo compose file does not run a recorder. Every other
`staging_dir` in the tree builds an `ArchiveWriterConfig` directly and never
reaches the mode dispatch.

`recorder_example.toml` gained the statement in prose. It is the archive-mode
configuration an operator copies, and a copy run without the flag is refused —
loudly and correctly, but the example should say which arrangement it is.

…te runs the ninth fault

Archive mode prints its mode line and then the plan; inline mode printed the
plan first and the mode arrived inside `config.summary()` further down. The
description argues that line is the one thing an operator must not miss — it is
what a command line can now get wrong by saying nothing — so an operator
scanning two hosts read it in two different places. `INLINE_MODE` moves out of
the summary and is printed before the plan in both branches, once.

Both tests read the *first line* rather than searching the output, in both
files, because `contains` is what let the two orders diverge without anything
failing.

And `Fault::SilentChannel` joins the equivalence gate's list, which is now all
nine. The rustdoc says which half of that fault the gate covers: the derivation
half, with one channel among several falling silent partway through. Not the
timing half — archive mode finds a quiet channel when a segment rotates on its
interval and inline mode when a window closes on age, and one window holding
the same datagrams as one segment is this fixture's whole premise. Asserting
that wants a fixture with a clock over several windows.

`SilentChannel` was missing from the list for the life of the branch and
nothing failed, because a fault absent from a loop is not a fault that fails —
it is a fault nobody runs, and that looks identical to a pass. Adding the ninth
fault closes the instance; this closes the class.

The list the gate runs and the list of every fault there is are kept separately,
because a list compared against itself compares nothing, and a `match` nobody
calls is what keeps the second one honest: a new `Fault` variant is a compile
error there. Dropping a fault from the gate's list now fails
`the_gate_runs_every_fault_the_replay_crate_injects` by name.

…had no mutant

Six more, each run and each restored. The one worth reading is the fault list:
dropping a fault from the equivalence gate left every test passing, because a
fault absent from a loop is not a fault that fails. That is how it went missing
for the life of the branch, so the list's completeness is now asserted.

Task 11 replaced its own mechanism rather than reversing its sentences, and
the replaced version is recorded rather than deleted: what survived is the
analysis of what silence must mean, what did not is the conclusion that
silence should mean a mode. The list of ten places the default was written
down becomes a list of the places a flag was, and every one loses it.

The selection lands in one place — `Arrangement::selected_by`, compiled in
every build, because a build that cannot run inline mode is precisely the
build that has to refuse a configuration selecting it. Its two refusals
move up out of the second file's error type, since a second reading of the
same two keys downstream of the selection is a reading that can disagree
with it.

A fifth test carries what four hand-written cases cannot: the four shapes
are total because the two predicates the selection reads are the two the
refusals name, and nothing about four separate tests says so.

Task 13 is the third review's four findings. The market data gap is
decided as not-derived with the ask existing in order to be refused, and
the plan records that this is the one answer in the task that is arguable,
with the alternative it was weighed against. The two spool findings and
the sort-key guard follow, each landing on its own.

…ag names it

The mode was a default with a flag over it. It is now a reading of two
predicates: an `[archive]` directory carrying a value, and a second file
given. Four combinations, total because both key sets are required with no
default and are disjoint — archive, inline, both refused, neither refused.

`Arrangement::selected_by` is the one place they exist, and it is compiled
in every build: a build without inline mode is precisely the build that has
to refuse a configuration selecting it, and a refusal that exists only
where it is not needed is no refusal. `Arrangement::Inline` therefore loses
its cfg, since that build now constructs the variant in order to refuse it,
and `writes_an_archive` becomes one comparison instead of a comparison and
a `true`.

The two refusals move up out of `InlineConfigError`, and
`check_archive_is_not_configured` is deleted rather than moved: its
condition is the selection, and a second reading of the same two keys
downstream of the selection is a reading that can disagree with it. Their
wordings change with their position. `ArchiveDirectoryConfigured` was a
migration instruction and becomes `BothArrangementsStated`, naming the key
and the file so both statements an operator has to reconcile are on the
screen. `NotStated` said "no mode was named, so this is inline mode" and
becomes `NoArrangementStated`, because no mode named is no longer a mode.
`NotCompiledIn` is now reached by a positive statement rather than by
silence, so it stops naming a flag and names the directories instead.

`--archive` is retired, not kept as an override. Every command line it
would be legal on is one the configuration already decides, and the only
power an override adds is the power to resolve a refusal — which is
starting a recorder in an arrangement its own configuration contradicts.
`a_flag_naming_an_arrangement_is_not_a_flag_this_binary_has` asserts it is
a usage error and that USAGE does not teach one, because a flag is cheap to
add back and the argument against it is long.

The migration is visible in the tests as the edit they did not need:
`check_mode.rs`, `shutdown.rs` and `inline_mode.rs` select archive mode
with the directories their fixtures already carried, on the command lines
they already ran. That is the fleet in miniature, and it is why this
replaced the inversion.

`the_four_configuration_shapes_are_total_and_two_of_them_run` holds the
thing four separate tests cannot: that the two predicates read are the two
the refusals name. A fifth predicate later would leave four case tests
green over a rule that had stopped being total.

…y tables answer nothing

Inline mode derives the five transport grains and none of the three market
data ones. That emptiness is not new and is not what this changes: archive
mode leaves `event`, `instrument` and `book_top` equally empty for a feed
with no `[[market_data]]` entry in the loader's configuration, and the
loader defends it there. What inline mode had that archive mode does not is
no way to ask and nothing saying so — three permanently empty tables,
indistinguishable from a feed nobody published on, and no key an operator
could have written to find out.

So the ask exists in order to be refused. `InlineConfig` gains the loader's
own `MarketDataFeed` rather than a second type — one spelling of which
feeds derive market data, so the two arrangements cannot grow two — and any
entry is `MarketDataNotDerived` at `--check`, naming the feed, the three
tables and archive mode. It is checked before the destination probe, so a
host learns it whether or not the column store is up.

Deriving them was considered and is a design rather than a task. Market
data derivation is a codec walk, and nothing in the record path decodes a
datagram — the rule that makes the transport grains trustworthy, since a
message a decoder would reject still carries the sequence number whose
absence is the finding. In archive mode that walk runs in the loader, off
the host, over bytes already stored and already hashed; on a recording
process's derivation stage a decoder that refuses costs the window, and
with it the transport rows the tier exists for. And `derive_events` builds
its instrument table per call, so per window nearly every price message
would be refused for an unresolved instrument unless the table, the book
and every open snapshot cycle were carried across windows and across a
restart — durable state spanning the unit the spool exists to bound.

A refusal only reaches an operator who tried, so the summary states it in
both arrangements: `market_data=none` beside the inline mode line, and
`market_data=loader` beside archive mode's, because there the process that
decides is the other one.

Not made a required key. That was considered and rejected: it would make
inline mode stricter than archive mode about a decision the two make
identically, and the equivalence of the two arrangements is what the gate
on this design asserts everywhere else. If the un-asked case should be
loud, that is a change to the loader too.

…eaves nothing the budget cannot see

Two spool findings from the third review, and a third in the guard that
was meant to catch a column reaching a sort key.

`enforce` selected `windows.keys().next()` with none of `take_oldest`'s
filter. Evicting an in-flight window is deliberate and unchanged — those
rows may yet land and no ledger entry has been earned. Evicting one that
*owes* an entry is not the same thing: its rows are in the store, and its
directory is the only thing left that can record that they are, carrying
the trailer the next era anchor is checked against. So it goes last,
preferred against while anything else remains and taken when nothing does,
because a budget that stopped bounding the disk to protect an entry would
trade a bounded backlog for an unbounded one.

And on a `remove_tree` failure it returned with the window still in the
map, so `bytes()` stayed over the budget for ever and the same undeletable
directory was chosen on every later pass — the budget stopping bounding the
disk from the first failure on. It now goes through `delete`, which takes
the window out of the map and moves its bytes to `unreclaimable_bytes`, and
*then* stops the pass. The pass still stops, for the reason it always did;
what changes is that the next pass tries the next window rather than the
same one.

`store` could leave a directory on disk and out of `self.windows`, so its
bytes sat outside `bytes()`, `enforce()` and `unreclaimable_bytes` alike:
the spool reported itself empty while orphans accumulated, one per failed
window, until a restart. The fallible part is now `write_window` and
`store` is its error path — a `?` in the middle of a function that has
already created a directory is a `?` that leaks it, and there were six of
them. The byte count is a claim about the disk, and a claim with an
exception is not one.

The sort-key guard read only the line beginning `ORDER BY`, and three of
the eight keys wrap, so `derivation` appended to a continuation line would
have passed. It now accumulates a clause until the parenthesis depth it
opened returns to zero — which also terminates a bare `ORDER BY` inside a
window specification on its own line. The clause count is asserted too: a
guard that reads more than one line is a guard whose reading is the thing
that can regress, and a walker that quietly stopped finding clauses would
leave it green over the hazard it was widened for.

…in both directions

Tasks 11 and 13 tick, and their ten reverts are recorded with the test that
died under each — committed first, mutated, watched fail, restored.

The sort-key one is recorded in both directions, because that is the only
way that finding can be shown. The widened guard fails on `derivation`
appended to a wrapped sort key; with the walker also reverted to the line
the clause starts on, the same hazard **passes**. That is the finding — not
that the guard could be better, but that it was green over a column in a
sort key.

Two weaknesses are stated rather than dressed up. The `market_data=none`
summary line is held only by a test asserting a string is printed, which is
the weakest shape of gate there is; the refusal beside it is what carries
the finding. And nothing asserts that a window merely in flight is still
evicted, because a test asserting the absence of a condition passes for as
long as nobody adds it.

…or on describes a run the capture stopped after

`window_seq: 0` and `preceding: None` keep their values and gain their reason.
The predecessor test is `segment_seq + 1` and a run begins at zero, so the
ledger's trailer precedes nothing in the run that reads it: wiring the read
alone changes no answer anywhere, and the version that continues the sequence so
that it would tells a reader the derivation was not down over the one interval
in which it was. `007_recorder_cross_site.sql` is what pays for that — the
capture-drop counter belongs to the capture handle, and a delta of zero over a
handle opened seconds earlier reads as a host that admitted nothing.

`NoLedger` loses the clause promising an era anchor certain across a restart,
which nothing keeps. The ledger is still required for the reason that is true.

`a_restart_does_not_anchor_its_first_window_on_the_ledgers_trailer` runs two
pipelines over one spool directory and one ledger, asserts the trailer is there
to be read, and asserts the run that starts under it still numbers its first
window zero and still writes an uncertain anchor.

…killed nothing

The mutation the two threads name — the trailer read on its own — leaves the
suite at 1457 passing, this task's own test included. The one beside it, which
also seeds the window sequence, kills that test on both assertions and prints
what it writes instead: an era row carrying anchor_certain 1 and continuation 1
over an interval in which nothing was captured.

…s captured

`dz-recorder-inline` linked `dz-recorder-capture` for exactly one type:
`PendingLoss`, which is `{ owed: u32 }` and touches nothing else. That
dependency does not build off Linux — `dz-recorder-capture` reads
`ScmTimestampns`, `RxqOvfl` and `Ipv4Ttl` out of nix's `ControlMessageOwned`,
and none of the three exists on macOS — so `cargo test -p dz-recorder-inline`
failed at the dependency and the crate's tests could not be run on a
developer's machine at all.

The argument for moving it is the one this crate's own Cargo.toml already
makes about `dz-recorder-replay` and `OwnedDatagram`: a crate that opens no
pcapng should not link a pcapng parser to name a type. This crate captures
nothing. It is handed datagrams and accounts for what it could not take, and
`PendingLoss` is that accounting, so it belongs in `dz-recorder-core` beside
`RecordedDatagram::drop_delta` — which is what its doc comment is about.

`dz-recorder-capture` re-exports it, as `dz-recorder-replay` re-exports
`OwnedDatagram`, so a capture still reads it where it reads everything else.
`dz-recorder-inline` no longer names the capture crate at all: `cargo tree -p
dz-recorder-inline -e normal,dev` contains neither it nor `pcap`. `nix` still
appears there through `ctrlc`, which is cross-platform.

The macOS build is not verified here — this host has no darwin target
installed — so what is claimed is the closure, which is checkable, and not
the compile.

No test moves and none is added: the workspace suite is 1457 passing before
and after. Gates: fmt; clippy on the default set, `--no-default-features`,
`-p dz-recorder-capture --features afpacket` — which is where the moved
import had to be fixed, since the default set does not compile that file —
and `-p dz-recorder-e2e --features socket-e2e`.

…fore the join it bounds

Three findings on the inline pipeline, all in `pipeline.rs`.

**One wall clock.** `pipeline.rs` carried a private `now_unix_nanos` while
`spool.rs`, in the same crate, imported `dz_recorder_load::now_unix_nanos`.
The two disagreed on overflow: the private one saturated, the shared one
truncated with `as u64`. The private copy is deleted and the surviving one
takes the saturating body, so nothing is lost by the deletion — a wrapped
stamp is a row dated inside the sequence space of every other row, which is
an ordering error rather than an out-of-range one and the harder of the two
to see.

**A restart is paced.** `remaining_after_stop` counts down only while the
stop flag is set, so a stage that panicked on every pass while *running*
restarted at the speed of the panic: a busy loop taking the CPU the capture
needs, writing a line to stderr per pass and burying the first one — the only
one that names the bug. The wait now doubles from `RESTART_DELAY` to
`RESTART_DELAY_MAX`, and a pass that ran longer than the ceiling resets it,
so a stage that panics once an hour is never slowed. The shutdown path is
deliberately not paced: it is bounded by the count instead, and sleeping
between attempts already given up on spends a supervisor's stop timeout.

**Each stage's flag is set before the join that waits on it**, which is why
there are now two. `stop` set one flag *between* the two joins, so a
derivation inside the restart loop was joined for ever — nothing counted
down, and the shutdown never reached the posting stage. That one flag could
not simply be moved earlier: setting it before the derivation's join would
tell the posting stage to make its last pass while the derivation was still
spooling into it, which is the ordering `stop` exists to get right.
`derivation_stage` reads its flag nowhere — the ring closing is its ending —
so the split costs nothing on the ordinary path.

Three tests, on `spawn_stage` directly, since the stages themselves offer no
panic injection point: every spool lock in this file is
`unwrap_or_else(PoisonError::into_inner)`, so a panicking sink cannot make a
second stage panic.

* `a_stage_that_panics_immediately_is_paced_before_it_is_begun_again` —
  three panics owe the floor, twice it and twice that. Remove the sleep and
  it finishes in microseconds and fails; the restart *count* is identical
  either way, which is why the assertion is on elapsed time.
* `a_stage_that_panics_on_every_pass_ends_once_its_flag_is_set` — waits for
  the thread rather than joining it, so a regression fails the test instead
  of timing the suite out with nothing in the log naming it.
* `giving_up_on_a_stopping_stage_costs_no_delay`.

**What has no test is the ordering itself.** Putting the flag back between
the joins passes every test in the workspace; failing it needs a panicking
derivation, which nothing here can produce. The unit test above pins the half
that is reachable — the loop ends only while its flag is set. Recorded here
rather than left for a reviewer to find by running the revert.

Gates: fmt; clippy on the default set, `--no-default-features`, `--features
afpacket`, `-p dz-recorder-e2e --features socket-e2e`. `cargo test
--workspace` 1457 passing before, 1460 after.

…eted

`8fb68c8` removed `--archive` and made each arrangement selected by the
configuration only it can run on. Three comments did not follow it, and each
tells a reader the opposite of what the code does.

`recorder_example.toml` contradicted itself inside seven lines: "there is no
flag" at the top, "The flag is not optional" below it. It now says what
selects an arrangement and what a host in the other one leaves unstated.

`tests/inline_mode.rs`'s module docs called inline mode the reading of a
command line naming no mode, and described two of its own tests as being
about a default. Neither arrangement is a default; the tests are about the
selection, and about `inline` being in the default feature set because one
released binary serves hosts in both arrangements.

`dz-recorder/Cargo.toml`'s `inline` feature said "a command line naming no
mode" fails without the feature. Every command line names no mode now. The
refusal fires on a configuration stating the inline arrangement, which is
what it says.

Comments only, and no behaviour: 1460 passing before and after. The
assertions that keep the flag gone — `a_flag_naming_an_arrangement_is_not_a_
flag_this_binary_has` and the `USAGE` check — were already there and are
untouched.